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ABSTRACT 

Glasgow Caledonian University Library (Scotland) 
conducted a pilot study to design a set of user chosen performance 
measures which can be used in British academic libraries for data 
collection. Members of 8 stakeholder groups were identified and given 
an 103-question survey, with each question to be rated on a 1~5 scale 
of importance. Stakeholder groups included: SPARC (Strategic Planning 
and Resources Committee); senior library staff; other library staff; 
academic staff; research students; postgraduate students; full-time 
undergraduates; and part-time undergraduates. Results from the pilot 
study indicate: the higher the academic status of the respondent, the 
more importance is attached to quality of staff and their management 
performance; and non-library staff respondents rated highly 
indicators which related to service provision, but rated technical 
and management issues lower. The final study will compile a 
meaningful list of stakeholders; identify a small number of generally 
applicable "pars imonious " measures; identify performance measures 
appropriate to specific stakeholder groups; allow decision-making 
librarians to concentrate on a defensible, validated set of measures; 
help non - librarians participate usefully in library performance 
measurement; and design both general and stakeholder-specific 
questionnaire skeletons which can be applied in all British 
university libraries. An appendix includes the pilot study 
questionnaire. (SWC) 
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Background 

G lasgow Caledonian University is one of 

Britain’s new universities and was created in 
1993 out of the former Glasgow Polytechnic and 
Queen’s College, Glasgow. The Library is spread 
over three campuses which, in order of size are: 

City Campus with 9,435 students (80%); Park 
Campus with 1,342 students (11%) and Southbrae 
with 1,082 students (9%). There are about 566 
teaching staff and 435 support staff. The model of 
administration is a centralised one, the university 
being administered by a University Management 
Group (UMG) consisting of the Principal, vice prin- 
cipals, assistant principals and secretary to the 
University Court. With the addition of the three 
deans of faculty this becomes the Strategic Planning 
and Resources Committee (SPARC). 

Glasgow Caledonian University Library has a 
policy of evaluating its services by a range of meth- 
ods and if the issue examined is of general concern 
we sometimes seek external funding to research it 
more widely. In recent years the library has con- 
ducted surveys on such issues as the needs of part- 
time students, research students and OPAC satisfac- 
tion. An annual general satisfaction survey was 
introduced in January 1994, using, on this first 
occasion, the survey model recommended by Van 
House (1990). 

The stakeholder project originated as part of the 
continuing programme of activity and interest in 
qualitative evaluation at Glasgow Caledonian 
University Library. A review of the work undertak- 
en, in 1993, showed that although it was useful an 
overall framework of performance issues, linking 
specific concerns, was needed. Previous experience 
suggested that these performance issues should 
spring from the needs and experiences of different 
categories of users and be used to facilitate data col- 
lection and decision-making (user orientated evalua- 
tion). This focused attention on the multiple con- 
stituencies model which defines effectiveness as the 
extent to which the needs of key constituencies are 
met. This originated with the work of Van House 
and Childers who split public library users into key 
constituencies, ie. groups of people who would be 



expected to influence decisions about the public 
library directly or indirectly. Questionnaires which 
contained a list of 61 items were administered to 
representatives of constituent groups and constituent 
members were asked to rate these 61 items or per- 
formance indicators on a scale of 1 (low) to 5 (high) 
importance (Childers and Van House, 1989). 

These methodologies have been adopted by 
Philip Calvert and Rowena Cullen in their work on 
library effectiveness in New Zealand. This related 
initially to the public library sector (Calvert and 
Cullen, 1992) but, more recently, has been applied 
to academic library management (the published 
results are still to appear). 

They identified six constituent or stakeholder 
groups: 

1. The resource allocators (vice-chancellors, deputy 
vice-chancellors and members of key planning 
and resource allocation committees) 

2. The decision makers - senior library staff 

3. The service deliverers - library staff 

4. Graduate students 

5. Undergraduate students 

6. Faculty (academic staff) 

The questionnaire was formed using 99 perfor- 
mance indicators and again each issue was rated on 
a 1-5 scale. 

The Pilot Study and its 
Methodologies 

C ontact was established with Philip Calvert and 
Rowena Cullen who readily agreed to their 
methods being adapted for use by Glasgow 
Caledonian University. Their questionnaire was cir- 
culated around the professional library staff for their 
comments and criticisms and the questionnaire was 
modified and extended slightly as a result. There 
were now 103 questions to be rated on a 1-5 scale. 

For the purposes of the study at Glasgow 
Caledonian University eight stakeholder groups 
were identified which reflected the experience of 
the university. 
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1. SPARC (Strategic Planning and Resources 
Committee, 10 members) 

2. Senior library staff 

3. Other library staff 

4. Teaching (academic staff) 

5. Research students 

6. postgraduate students 

7. Full-time undergraduates 

8. Part-time undergraduates 

Administrative/support staff were not included 
because they do not make much use of library ser- 
vices or impact significantly on their provision. The 
need to break down students into four categories 
reflected past experience. In Glasgow Caledonian 
University, for example, part-time students com- 
prise about a third of the total. 

The actual administration and analysis of the 
questionnaire became the subject of a mathematics 
honours project and was undertaken by Siobhan 
Friel, a BSc Mathematics for Business Analysis 4th 
year Honours student. Before going to university 
she had worked as a university library assistant 
The groups she sampled were as follows: The popu- 
lations of the first three constituencies ie. SPARC, 
senior and other library staff were so small (9, 8 and 
35) that all members were sampled, giving response 
rates of 55.56%, 62.5% and 74.29% respectively. 
Academic staff were surveyed by stratified sam- 
pling. The staff list in the internal telephone directo- 
ry was used as a population list. The population was 
stratified by department and then simple random 
sampling was performed within each strata to get an 
overall sample of 176 staff from 22 departments. 
This generated a response rate of 28.41%. Research 
students were sampled systematically. Every second 
student received a questionnaire through the internal 
mail, giving a sample of 108 with a response rate of 
19.46%. 

Postgraduate, full-time undergraduate and part- 
time students were quota sampled on the basis of 
faculty: Business, Health and Science and Tech- 
nology. Full-time undergraduates were further sub- 
divided by year of study. Postgraduate and part- 
time students were surveyed to give samples of 40 
each. Some 216 full-time undergraduates were sam- 
pled. 

The total sample size was 632. The data was 
analysed using Microsoft Excel. 
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Results 

R anked outputs were produced, both generally, 
and for specific stakeholder groups. There were 
eight indicators contained in the ‘Top 20’ of each 
constituency: 

1. Helpfulness, courtesy of staff 

2. Quietness of study environment 

3. Availability of sought material on shelf 

4. Expert staff assistance to users available when 
needed 

5. Expertise of reference staff 

6. Provision made for disabled users 

7. Total amount of library budget 

8. Equipment (eg. photocopiers) kept in service by 
good maintenance 

The word ‘staff’ appears three times. This list has 
been used to devise a new general satisfaction sur- 
vey questionnaire which was administered in 
January-February 1995 and proved easier to analyse 
and interpret than that based on Van House. 

There were, however, important variations 
between stakeholder groups. Detailed comparative 
rankings of these were not produced and this will 
have to be addressed in the main study. Senior and 
other library staff agreed about what was most 
important: themselves. They identified Helpfulness, 
courtesy of staff as the most important indicator. 

Four categories: academic staff, postgraduate stu- 
dents, undergraduate students and part-time students 
selected Quietness of study environment as being the 
most important. SPARC selected ‘Match of opening 
hours to user needs’ as its most important indicator, 
while the indicator ranked highest by research stu- 
dents was Reciprocal access to other university 
libraries. 

The indicator Competence of library management 
was considered important by all stakeholder groups, 
except full-time and part-time students. Part-time 
students was the only stakeholder group which did 
not rate Provision of multiple copies of items in high 
use as important enough to be in their ‘top 20’, per- 
haps an indication of their ability to obtain key texts 
by other means (Friel, 1994). Figure 1 lists the top 
five indicators (as identified by all categories) and 
shows how each category valued them. For satisfac- 
tory indicator evaluation more detailed work will be 
necessary. 

Even a cursory glance at a small number of indi- 
cators shows interesting variations. The senior 
library staff rate the noise issue lower than any other 
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Figure 1 







The Top Five Issues (Indicators) 






Issues 


AU 


Sparc FTU 


PTU 


PG RES Acad 


SLS 


OLS 


Quiet 


1 


5 1 


1 


1 12 1 


14 


5 


Help 


2 


2 3 


9 


8 16 8 


1 


2 


Multi 


3 


43 2 


35 


22 5 4 


5 


18 


Assist 


4 


14 8 


10 


5 9 5 


32 


4 


Avail 


5 


3 5 


17 


11 7 7 


16 


27 








Keys to the table 








Stakeholders 




Issues 








All - 


all categories combined 




Quiet - quietness of study environment 






Sparc - 


SPARC 




Help - helpfulness , courtesy of staff 






FTU - 


full-time undergraduates 




Multi - provision of multiple copies of 






PTU - 


part-time undergraduates 




items in high use 








PG - 


postgraduate students 




Assist - expert staff assistance to users 






RES - 


research students 




available when needed 








Acad - 


academic staff 




Avail - availability of sought materials on 






SLS - 


senior library staff 




shelf 








OLS - 


other library staff 












category, despite its obvious importance, perhaps a 
reflection of the limited success in combating it 
over many years. Research students also do not rate 
it highly but research students have their own office 
accommodation. Academics rate it the most impor- 
tant issue which, presumably, shows them identify- 
ing with their undergraduate students. Provision of 
multiple copies of items in high use shows the acad- 
emics (and senior library staff) identifying with full- 
time students while SPARC, from a remoter per- 
spective, consider it relatively unimportant. 

Overall, two general points seemed to emerge: 

1. The higher the academic status of the respondent 
in the university, the more importance is attached 
to quality of staff and their management perfor- 
mance. 

2. Non-library staff respondents, on the whole, rated 
highly indicators which related to service provi- 
sion. Technical and management issues they rated 
lower, presumably taking the view that technical 
or purely professional librarianship issues were of 
less concern to them. 

Methodological Conclusions 

1 Clearly the questionnaire, with its 103 perfor- 
• mance issues was too long. Some measures 
were given a low rating by all stakeholder groups. 



2. The terminology of some of the measures was 
clumsy. Poor terminology needed to be improved 
and clumsy wording needed to be re-examined to 
see if the measure itself is inappropriate. 

3. Some measures appeared to overlap or duplicate 
themselves. 

4. Service based measures may be more appropriate 
than technical ones. 

5. The measures are somewhat dated and take little 
account of electronic networking and electronic 
information services. 

6. Recent work such as The effective academic 
library (Joint . . , 1995) and the current research 
work summarised in it need to be taken into 
account. 

7. Detailed comparative analysis of the different 
stakeholder groups is needed. 

8. The random order arrangement was perceived by 
some to be unhelpful. 

The Main Project 

O n the basis of the pilot study an application for 
research funding was made to the British 
Library Research & Development Department in 
July 1994 to employ a research assistant for a year 
to administer a modified version of the question- 
naire, both at Glasgow Caledonian University and 
other universities. 
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The overall aim of the project is to design a set of 
user chosen performance measures which can be 
used in British academic libraries for data collection 
and decision-making 
The specific aims are: 

1. to identify a set of user chosen performance mea- 
sures 

2. to compile a meaningful list of stakeholders 

3. to identify a small number of generally applicable 
‘parsimonious’ measures 

4. to identify performance measures appropriate to 
specific stakeholder groups 

5. to allow decision-making librarians to concen- 
trate their attention on a defensible, validated set 
of measures 

6. to help non-librarians to participate usefully in 
library performance measurement 

7. to design questionnaire skeletons which can be 
applied in all British university libraries for 

a) a short, general satisfaction questionnaire, 
using the set of parsimonious measures arrived at 

b) specific stakeholder groups. 

The project was approved at the end of January 
1995 and partners were immediately sought with 
considerable success. The research assistant, Helen 
Pickering, who had previously worked at LISU, was 
appointed in June and began work at the beginning 
of July. The project is in three main phases 

1. July-September: formalising contacts with partic- 
ipants, redesigning questionnaires in conjunction 
with participants and planning operational 
methodologies. 

2. October-December 1995: Questionnaire adminis- 
tration will take place and data input will begin. 

3. January-June 1996: Data input will be completed, 
analysis and writing up and questionnaire design 
will be done. 

Progress to Date 

I We now have 15 participants, mainly in univer- 
• sides founded after 1960. 

2. There has been surprisingly little argument about 
stakeholder groups. The only needed additional 
category seems to be ‘research fellow’ or equiva- 
lent (ie. someone not pursuing a higher degree, 
but mostly working on research). 

3. The main concern among partners seems to be 
about sample sizes and the labour costs associat- 
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ed with administering the questionnaire. This 
reflects a common paradoxical attitude among 
library managers. They are anxious to collect, 
both quantitative and qualitative data about their 
services, but fear the costs involved. In planning 
sampling, which is being done with the help of 
the university’s Departments of Economics and 
Mathematics, we hope to learn some lessons 
which will allow reliable and realistic sampling at 
an acceptable cost. Undergraduate populations 
are the main problem and here, systematic ran- 
dom sampling seems the only solution. 

Redesigning the Questionnaire 

F ollowing criticisms about the random structure 
of the questionnaire the measures have been cat- 
egorised into six groups into which they appeared 
naturally to fit. The groupings proposed in The 
effective academic library (JFC, 1995) were consid- 
ered but were found to be inappropriate. The six 
groups are as follows 

1. Management and administration 

2. Equipment / computer hardware 

3. Staff 

4. Stock maintenance and provision 

5. Physical library features/environment 

6. Users, services to users and service usage. 

Overall analysis of each group will be possible and 
it will be possible to relate stakeholder groups to 
specific categories. 

Outcomes 

'■J^he main outcomes of the project will be 

1. a report to BLR&DD 

2. sets of performance measures, both general and 
those appropriate to particular stakeholder groups 

3. a set of questionnaires ie. a general one and ques- 
tionnaires appropriate to each stakeholder group 

4. advice on methodologies for questionnaire 
administration and costing in British university 
libraries 

5. every effort will be made to report on the work at 
conferences and other fora. 

The overall impression seems to be that a standardi- 
sation of concerns is taking place in British univer- 
sity libraries and, if this is the case, then the impli- 
cations are very wide. 
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Appendix 1 

Pilot Study Questionnaire 



GLASGOW CALEDONIAN UNIVERSITY 
PERFORMANCE MEASUREMENT PILOT STUDY 

This is a list of criteria which might be used to judge the performance of a university library. 
Which of these seem, from your point of view, more important and which least important in 
judging the performance of a university library? 

Put a circle around one number on each line. Circle 5 if you think the item is most important. 
Circle 4 if it is not quite so important, and so on. Circle 1 for the items you think have the least 
importance. 

Please remember, you are not judging the performance of your own university library, but 
saying what criteria you would like to see used to measure the performance of a university 
library. 

Regular evaluation of building 
Facility for users to recommend items for purchase 
Access to library catalogues via networks throughout campus 
Expert staff assistance to users available when needed 
Frequent stock editing procedures 
Competence of library management 
Proportion of staff professionally qualified 
Regular communication with user groups 
Level of staff workload 
Speed of acquisition of new materials 
Availability of library catalogues throughout the library 
Conservation principles used in housing library materials 
Match of goals and objectives to user group needs 
Provision made for disabled users 
Expertise of reference staff 

Number of library staff per full-time equivalent academic staff 
Provision of personal computers for users’ own work 
Availability of sought material on shelf 
Speed of provision of items through inter-library loans 



MOST LEAST 
IMPORTANT IMPORTANT 

5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 
.5 4 3 21 
5 4 3 2 1 



Page 1 





MOST 

IMPORTANT 


LEAST 

IMPORTANT 


Openness of management procedures and documents to users 


5 


4 


3 


2 


i 


Equitable allocation of materials budget amongst subjects taught 


5 


4 


3 


2 


i 


Availability of reference staff when needed 


5 


4 


3 


2 


i 


State of repair of materials (books, journals etc.) 


5 


4 


3 


2 


i 


Use of collection development policies 


5 


4 


3 


2 


i 


Proportion of library budget spent on materials 


5 


4 


3 


2 


i 


Distance of library from teaching areas 


5 


4 


3 


2 


i 


Division of library materials expenditure between books and 


periodicals 


5 


4 


3 


2 


1 


Arrangement of library collection 


5 


4 


3 


2 


1 


Facility to reserve items on short loan 


5 


4 


3 


2 


1 


Provision of adequate number of photocopiers 


5 


4 


3 


2 


1 


Proportion of materials budget spent on research materials 


5 


4 


3 


2 


1 


Adequate and pleasant workspace for staff 


5 


4 


3 


2 


1 


Feedback to users who request items for purchase 


5 


4 


3 


2 


1 


Total number of registered borrowers 


5 


4 


3 


2 


1 


Match of open hours to user needs 


5 


4 


3 


2 


1 


Total number of items held by library 


5 


4 


3 


2 


1 


Extent to which services are free 


5 


4 


3 


2 


1 


Total number of items borrowed per year 


5 


4 


3 


2 


1 


Range of types of material (videos, computer software etc.) 


5 


4 


3 


2 


1 


Number of seats per full-time student equivalent 


5 


4 


3 


2 


1 


Quietness of study environment 


5 


4 


3 


2 


1 


Proximity of refreshment service during library open hours 


5 


4 


3 


2 


1 


Provision of multiple copies of items in high use 


5 


4 


3 


2 


1 


Provision of microfilm and microfiche readers 


5 


4 


3 


2 


1 


Library staff involvement in organisational life of university 


5 


4 


3 


2 


1 


Provision of group study rooms 


5 


4 


3 


2 


1 


Equitable and effective fines policy 


5 


4 


3 


2 


1 


Provision of photocopiers in all areas of the library 


5 


4 


3 


2 


1 


Equipment (e.g. photocopiers) kept in service by good maintenance 


5 


4 


3 


2 


1 


Access to CD-Roms and databases via networks throughout the 


campus 


5 


4 


3 


2 


1 


Use of planning procedures (short and long term) 


5 


4 


3 


2 


1 


Number of seats occupied at peak hours 


. 5 


4 


3 


2 


1 
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MOST 

IMPORTANT 



LEAST 

IMPORTANT 



Proportion of library materials listed on computer catalogue 
Amount of use of materials in library without being borrowed 
Total amount of library budget 
Display of new books and periodical issues 
Availability of printed periodical indexes 
Currency of library materials 
Safeguards against mutilation and theft 
Equitable allocation of materials budget between groups of 
users (staff/students) 

Speed of recall of items out on loan requested by other users 
Proportion of journals bound as opposed to unbound 
Efficiency / cost effectiveness of library 
Availability on shelf of items listed in catalogue 
Library expenditure per full-time equivalent student 
Percentage of collection borrowed each year 
Number of library staff per full-time equivalent student 
Success rate in answering reference questions 
Provision of personal study carrels 
Amount of total library budget as proportion of university 
expenditure 

Availability of all library collections for browsing 
Extent to which library achieves goals and objectives 
Adequacy of library collection compared with other institutions 
Speed of recall of reserved items 

Amount of user education (i.e. teaching use of library and materials) 

Extent of involvement of user groups in decision making 

Ease of use of public catalogues 

Proportion of total stock restricted to short term loan 

Provision of teaching facilities within library 

Comfort, appeal of building 

Flexibility of budget to respond to new subject areas 
Regular notification of users of new materials added to stock 
Proportion of library budget spent on staff 
Extent to which users are made aware of services available 
Availability of user seating near reference collection 



5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 

5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 

5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 



1 

1 

1 
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MOST 

IMPORTANT 



LEAST 

IMPORTANT 



Speed and accuracy of re-shelving of materials 
Cost per item (books) added to stock 
Percentage of stock not used in past five years 
Amount of time journals are out of circulation for binding 
Rate at which collection is growing 
Number and quality of signs for direction / guidance 
Range of library services available whenever library is open 
Number and quality of written management policies 
Availability of periodical indexes on CD-Rom 
Percentage of potential users actively using the library 
Availability of user-pays online searching of periodical indexes 
Helpfulness, courtesy of staff 
Proportion of items wanted by user finally obtained 
Flexibility of loan periods 
Staff training and development 
Number of items borrowable at one time 
Reciprocal access to other university libraries 
Cost of photocopying 

Thank you for participating in this pilot research project. 

Please return this completed questionnaire to any site Library, put in the internal mail, or post 
to either undemoted address: 

Siobhan Friel 
BSMB4(Hons) 

c/o Maths Dept. Office, W619A 
City Campus 

Glasgow Caledonian University 
Cowcaddens Road 
GLASGOW 
G4 OBA 

Category: 



John Crawford 
Library Research Officer 
The William Harley Library 
City Campus 

Glasgow Caledonian University 
Cowcaddens Road 
GLASGOW 
G4 OBA 



5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
5 4 3 2 
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